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The  phrase  “testing  in  a  joint  environment"  refers  to  testing  military  weapons  and  supporting 
systems  in  the  joint  mission  environments  in  which  those  weapons  and  systems  are  expected  to 
operate.  The  Office  of  the  Secretary  of  Defense  chartered  the  Joint  Test  and  Evaluation 
Methodology  project  to  institutionalize  testing  in  a  joint  environment  by  improving  the  ability 
to  conduct  tests,  across  the  acquisition  life  cycle,  in  realistic  joint  mission  environments. 
Specifically,  the  project  was  directed  to  develop  methods  and  processes  for  using  distributed  live- 
virtuaTconstructive  joint  test  environments  to  evaluate  system  performance  and  joint  mission 
effectiveness.  In  2007,  the  project  completed  a  series  of  such  tests  to  assess  an  initial  set  of 
methods  and  processes.  Tests  of  network-enabled  air-to- surface  weapons  and  ground-launched 
surface-to-surface  precision  attack  missiles  were  used  to  provide  context  for  system  performance 
evaluations.  Joint  mission  effectiveness  was  evaluated  by  conducting  Joint  Fires  and  Joint  Close 
Air  Support  with  the  above  weapons  and  other  supporting  systems.  These  tests  were 
accomplished  as  part  of  the  2007  INTEGRAL  FIRE  event  sponsored  by  the  Air  Force 
Integrated  Collaborative  Environment  program.  This  article  describes  results  after  methods  and 
processes  for  testing  in  a  joint  environment  were  used  by  experienced  testers  to  design  and 
assemble  an  actual  distributed  joint  test  environment.  Results  identified  improvements  to  the 
processes  as  well  as  recommendations  for  test  organizations.  To  streamline  routine  test  planning 
for  distributed  testing,  we  recommend  test  organizations  consider  procedures  such  that  each 
acquisition  program  has  a  lead  test  organization  designated  for  distributed  testing.  We  also 
recommend  that  test  organizations  consider  establishing  formal  relationships  to  manage  the 
distributed  test  environment  as  a  single  facility. 

Keywords:  Testing  in  a  joint  environment;  distributed  testing;  live-virtual-constructive; 
LVC;  joint  test  environment. 


The  Joint  Test  and  Evaluation  Method¬ 
ology  QTEM)  project  was  chartered  to 
investigate,  evaluate,  and  make  recom¬ 
mendations  to  improve  the  ability  to 
test  across  the  acquisition  life  cycle  in 
realistic  joint  mission  environments.  Specifically, 
JTEM  was  charged  with  developing,  testing,  and 
evaluating  methods  and  processes  for  defining  and  using 
a  distributed  Live,  Virtual,  and  Constructive  (LVC) 
joint  test  environment  to  evaluate  system  performance 


and  joint  mission  effectiveness.  JTEM’s  initial  set  of 
methods  and  processes  were  developed  in  2006.  In  2007, 
the  INTEGRAL  EIRE  test  event  was  used  to  evaluate 
those  methods  and  processes,  which  were  used  to  plan 
and  conduct  tests  of  two  systems  as  participating 
elements  in  an  overarching  system  of  systems.  In  this 
article,  we  describe  those  tests  and  how  JTEM  methods 
and  processes  were  used  to  plan  and  execute  them.  The 
section  ‘Testing  in  a  joint  environment”  provides  some 
background  information  on  the  vision  within  the 
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Department  of  Defense  for  improved  testing  in  defense 
acquisition,  including  the  notion  of  testing  in  a  joint 
environment  and  the  genesis  of  the  JTEM  project.  The 
section  “Methods  and  processes  for  testing  in  a  joint 
environment”  describes  the  specific  methods  and 
processes  used  during  INTEGRAL  FIRE  planning 
and  execution.  Sections  “Applying  the  methods  and 
processes”  and  “Results  and  discussion”  explain  system 
performance  and  joint  mission  effectiveness  tests  result¬ 
ing  from  the  appUcation  of  JTEM’s  methods  and 
processes.  The  “Results  and  discussion”  section  explains 
the  ability  to  evaluate  the  results  of  the  systems 
performance  and  joint  mission  effectiveness  tests  as  an 
indication  of  the  effectiveness  and  suitability  of  JTEM’s 
methods  and  processes.  The  final  section  summarizes 
our  conclusions  and  recommendations. 

Testing  in  a  joint  environment 

What  is  testing  in  a  joint  environment?  Why  is  it 
important?  The  short  answer  to  both  questions:  to  test 
as  we  fight.  For  most  of  the  twentieth  century,  the  U.S. 
Air  Force,  Army,  Navy,  and  Marines  fought  wars 
together  by  coordinating  separate  air,  land,  and  sea 
operations.  Such  separate  operations  preserved  tradi¬ 
tional  service  roles  but  did  not  always  take  advantage  of 
synergies  among  service  capabilities.  Starting  in  1991, 
with  Operation  Desert  Storm,  and  continuing  through 
today’s  operations  in  Afghanistan  and  Iraq,  command¬ 
ers  from  one  service  have  been  compelled  by  circum¬ 
stances  to  conduct  operations  jointly  with  other 
services.  While  such  joint  operations  have  clearly 
proven  to  be  more  effective  than  separate  service 
operations,  joint  operations  also  reveal  incompatibili¬ 
ties  of  individual  service  systems  (hardware,  software, 
or  procedures)  with  one  another.  To  eliminate 
incompatibilities  in  future  systems,  the  Secretary  of 
Defense  changed  the  way  new  military  systems  are 
justified,  developed,  and  tested.  This  new  requirements 
initiation  system  (Department  of  Defense,  2003a)  uses 
a  capabilities-based  approach  to  identify  gaps  in  the 
Services’  ability  to  carry  out  joint  missions.  The 
Services  must  identify  new  systems  to  fill  the  gaps 
and  must  test  those  systems  to  determine  whether  they 
can  support  joint  operations.  Testers  will  need  joint 
environments  in  which  to  conduct  such  tests. 

In  his  strategic  planning  guidance  for  2006  to  2011, 
the  Secretary  of  Defense  directed  his  staff  to  determine 
what  actions  would  be  necessary  to  create  new  joint 
testing  capabilities  and  to  institutionalize  the  evalua¬ 
tion  of  joint  mission  effectiveness.  The  resulting 
Testing  in  Joint  Environment  Roadmap  (Department 
of  Defense  2004)  identifies  policy,  procedures,  and  test 
infrastructure  changes  that  would  allow  the  services  to 
routinely  conduct  test  and  evaluation  in  joint  environ¬ 


ments.  Parallel  policy  changes  require  frequent  testing 
of  all  systems  to  demonstrate  joint  capabilities 
throughout  development.  Procedural  changes  adjust 
the  traditional  methods  and  processes  testers  use  to 
define  test  environments,  design  test  events,  determine 
measurement  requirements,  and  establish  evaluation 
products.  Infrastructure  changes  are  needed  to  over¬ 
come  facility  and  force-availability  limitations.  Large 
forces  are  seldom  available  to  participate  in  testing 
because  of  real-world  commitments.  Even  if  forces 
were  available,  most  test  facilities  are  simply  too  small. 

Authors  of  the  Testing  in  Joint  Environment  Road¬ 
map  quickly  concluded  that  testing  in  joint  environ¬ 
ments  was  generally  not  possible  at  any  single  test 
facility.  They  saw  modern  networks  and  rapidly 
improving  simulations  as  the  means  to  overcoming 
single-facility  limitations.  Networks  can  make  several 
different  and  geographically  separated  test  facilities 
appear  as  one.  Networks  also  allow  operator-  or 
hardware-in-the-loop  simulators  (sometimes  called 
“virtual”  simulations)  to  substitute  for  live  systems, 
and  digital  computer  simulations  (called  “constructive” 
simulations)  to  substitute  for  live  or  virtual  systems  in  a 
joint  environment.  Combinations  of  live,  virtual,  and 
constructive  systems — linked  through  networks  into  a 
single  distributed  environment — could  form  LVC  joint 
mission  environments  for  testing.  An  added  benefit  is 
that  system  developers  can  test  early  constructive 
models  in  an  LVC  joint  mission  environment.  Those 
developers  can  continue  to  use  the  same  environment 
for  testing  virtual  and  live  prototypes  as  development 
work  progresses  toward  production.  Roadmap  authors 
see  the  LVC  mission  environment  as  a  key  enabler  to 
“testing  as  we  fight”  across  the  acquisition  life  cycle. 

Traditional  tests  conducted  by  the  military  services 
have  focused  on  verifying  system-level  performance 
requirements  specified  in  operational  requirements 
documents.  The  military  services  have  litde  experience 
testing  new  systems  as  participating  elements  in  a  joint 
system  of  systems.  As  a  result,  processes  and  methods  for 
designing  and  executing  tests  of  systems  of  systems  in 
joint  mission  environments  are  neither  well  defined  nor 
understood.  Nor  is  there  a  clear  understanding  of  how  to 
assess  system  performance  as  it  pertains  to  capabilities 
supporting  joint  missions.  The  Director  of  Operational 
Test  and  Evaluation  (DOTScE),  as  the  lead  Secretary  of 
Defense  staff  agency  for  the  Roadmap  and  its  imple¬ 
mentation,  chartered  JTEM  to  address  the  methods- 
and-process  components  of  implementation. 

Methods  and  processes  for  testing  in  a 
joint  environment 

The  initial  set  of  methods  and  processes  developed 
by  JTEM,  and  evaluated  during  INTEGRAL  FIRE,  is 
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Figure  1.  Capability  Test  Methodology  version  1.0  used  during  INTEGRAL  FIRE. 


called  the  Capability  Test  Methodology  (CTM) 
(Department  of  Defense,  2007a)  because  it  goes 
beyond  individual  systems.  CTM  is  the  foundation 
for  templates,  handbooks,  and  other  best  practice 
guidance  JTEM  will  ultimately  deliver  to  test  organi¬ 
zations  and  acquisition  program  managers  regarding 
testing  in  a  joint  environment.  Figure  1  shows  the  five 
steps  and  eleven  processes  of  which  the  CTM  is 
composed.  Of  course,  this  serial  depiction  is  a 
simplification  of  what  occurs  in  practice.  Most  CTM 
processes  are  iterative  in  nature;  many  are  performed  in 
parallel,  and  outputs  are  fed  back  into  other  processes. 

Nine  (out  of  eleven)  processes  were  the  focus  during 
INTEGRAL  FIRE.  These  nine  processes,  in  CTM 


steps  1  through  3,  are  important  to  the  design  and 
execution  of  systems  of  systems  tests  in  a  distributed 
LVC  joint  environment.  JTEM  did  not  develop  step  4 
processes.  The  other  two  processes,  in  CTM  step  5, 
deal  with  evaluations  of  system  performance  and  joint 
mission  effectiveness.  These  were  addressed  during 
2008.  Table  1  shows  the  first  three  CTM  steps  and  the 
various  output  products  produced  by  the  processes. 
These  output  products  were  used  to  assemble  the 
particular  distributed  live,  virtual,  and  constructive 
joint  mission  environment  for  INTEGRAL  FIRE. 

The  first  two  CTM  steps  are  derived  from  the 
current  processes  for  planning  tests  at  a  single  Major 
Range  and  Test  Facility  Base  (MRTFB)  (Department 


Table  1.  Primary  CTM  steps,  processes,  and  output  products  evaluated  during  INTEGRAL  FIRE 


CTM  steps 

CTM  methods  and  processes 

Output  products 

CTM  1  Characterize  test 

•  Develop  test  concept 

•  Develop  joint  operational  context  for  test 

•  Develop  evaluation  strategy 

•  Technical  assessment 

•  Program  Introduction  Document  (PID) 

•  Statement  of  Capability  (SOC) 

CTM  2  Plan  test 

•  Develop  test  design 

•  Perform  LVC  DE  analysis 

•  Develop  test  plan 

•  Test  plan 

CTM  3  Implement  LVC  distributed 

•  Design  LVC  DE  configuration 

•  Joint  Mission  Environment  (JME)  foundation 

environment 

•  Integrate  LVC  DE 

model 
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of  Defense  2002)  location.  Early  planning  negotiations 
between  distributed  test  organizations  and  their 
customers  (typical  program  managers,  for  example) 
are  conducted  during  CTM  step  1,  Characterize  Test. 
Program  characterization  processes  conducted  by 
customers  include  development  of  joint  operational 
contexts  for  tests,  development  of  test  concepts,  and 
development  of  the  evaluation  strategies.  Test  capabil¬ 
ity  characterization  processes  are  conduced  by  test 
organizations.  These  processes  include  technical  as¬ 
sessments  that  produce  initial  estimates  of  distributed 
test  facilities  needed  to  implement  test  concepts  and 
programmatic  assessments  that  produce  initial  schedule 
and  cost  estimates.  Program  Introduction  and  State¬ 
ment  of  Capability  Documents  produced  by  CTM  1 
follow  formats  defined  by  the  Range  Commanders’ 
Council  (Department  of  Defense  1997).  During  CTM 
2,  the  test  planning  phase,  test  concepts  are  developed 
into  more  detailed  test  plans.  Test  planning  processes 
include  designing  distributed  tests  in  joint  environ¬ 
ments;  refining  LVC  distributed  test  environments; 
and  synthesizing  these  activities  into  overall  test  plans. 
In  CTM  version  1.0,  we  assume  that  program 
introductions,  statements  of  capability,  and  test  plans 
reflect  the  requirements  of  a  single  customer. 

Joint  mission  environments  are  assembled  and  used 
to  support  multiple  test  plans  (e.g.,  customers)  during 
CTM  steps  3,  4,  and  5.  Implement  LVC  Distributed 
Environment  processes  are  concerned  with  technical 
systems  engineering  activities  for  automatic  distributed 
LVC  implementation.  These  processes  include  the 
design  of  distributed  configurations,  assembly  of 
distributed  components,  and  integration  of  compo¬ 
nents  into  a  distributed  LVC  “test  range”  that  meets 
customer  requirements.  In  CTM  step  4,  the  Execute 
Test  phase,  tests  are  conducted  according  to  procedures 
and  data  are  collected.  Schedules  are  developed  and  test 
events  are  run  using  test  planning  products  as  inputs. 
This  phase  produces  test  data  for  customers  and 
reusable  information  for  future  joint  mission  environ¬ 
ments.  Though  joint  mission  environments  are  assem¬ 
bled  to  support  multiple  customers,  tests  do  not  have  to 
be  run  concurrently.  Sometimes,  individual  customers 
may  separately  schedule  only  those  parts  of  the  joint 
mission  environment  they  need  to  meet  their  own 
objectives  for  testing  in  a  joint  environment.  Other 
times,  multiple  customers  may  share  a  joint  mission 
environment  at  the  same  time,  for  convenience  or  as  a 
result  of  hard  programmatic  requirements.  The  latter 
situation  was  assumed  during  INTEGRAL  FIRE.  In 
the  final  step.  Evaluate  Test,  data  are  processed, 
analyzed,  and  evaluated.  These  processes  turn  test  data 
into  knowledge  of  what  happened  during  tests, 
including  evaluations  of  joint  mission  effectiveness 


and  the  contributions  of  individual  systems  to  joint 
missions. 

Applying  the  methods  and  processes 

We  used  the  2007  INTEGRAL  FIRE  event  to 
develop,  test,  and  evaluate  JTEM  methods  and 
processes  when  those  processes  were  used  by  typical 
test  organizations  under  operationally  representative 
conditions.  INTEGRAL  FIRE  (Department  of  De¬ 
fense  2007b)  was  a  joint  capability  integration  event 
intended  to  support  joint  test  activities  while  working 
to  establish  persistent  joint  test  environments.  The 
event  was  jointly  sponsored  by  Secretary  of  the  Air 
Force,  Warfighter  Integration  Directorate  (SAF/XC); 
U.S.  Joint  Forces  Command,  Joint  Systems  Integration 
Command  (JSIC);  the  Joint  Mission  Environment 
Test  Capability  (JMETC)  program;  and  JTEM.  SAF/ 
XC  conducted  an  assessment  of  the  Warplan-to- 
Warfighter  Forwarder  System  and  its  ability  to  support 
dynamic  targeting.  JSIC  conducted  a  technical  assess¬ 
ment  of  digital  interoperability  during  the  processing 
of  immediate  requests  for  Joint  Close  Air  Support.  The 
JMETC  program  coordinated  network  connectivity 
and  middleware  for  assembling  the  joint  test  environ¬ 
ment.  JTEM  test  activity  provided  context  in  which  to 
apply  the  CTM.  INTEGRAL  FIRE  was  coordinated 
through  the  Air  Force  Integrated  Collaborative 
Environment  program.  Event  management,  led  by 
the  U.S.  Air  Force  Simulation  and  Analysis  Facility, 
was  conducted  collaboratively  across  several  distributed 
test  organizations.  The  test  organizations  that  sup¬ 
ported  JTEM  activity  are  shown  in  Figure  2. 

The  particular  test  activity  planned  and  conducted 
with  the  CTM  was  intended  to  represent  typical 
testing  in  a  joint  environment  during  early  system 
development.  As  such,  it  was  assumed  the  overall 
testing  in  a  joint  environment  objective  was  to  evaluate 
the  contributions  of  two  developmental  weapon 
systems  to  joint  mission  effectiveness  when  those 
weapon  systems  were  employed  together  as  participat¬ 
ing  elements  in  an  overarching  system  of  systems. 
Contributions  to  joint  mission  effectiveness  would 
then  be  used  to  determine  which  of  the  tested  system 
design  alternatives  warranted  further  development. 
Constructive  models  of  a  surface-to-surface  fire 
support  platform  (FSP)  and  an  air- to -surface  net- 
work-enabled  weapon  (NEW)  were  used  to  represent 
the  two  developmental  systems.  Joint  Fire  Support 
(Department  of  Defense  2006),  including  aspects  of 
Joint  Close  Air  Support  (Department  of  Defense 
2003b),  was  chosen  as  the  joint  mission.  We  assumed 
joint  mission  effectiveness  was  determined  by  the 
ability  to  deny  employment  of  enemy  forces  (timeliness 
of  attacks)  and  the  ability  of  the  system  of  systems  to 
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attack  enemy  combat  assets  (continuity  of  target 
location  accuracy  across  network  nodes).  Weapon 
design  alternatives  were  defined  by  different  data  link 
message  implementations.  Different  joint  tactics, 
techniques,  and  procedures  were  used  to  evaluate  the 
robustness  of  design  alternatives  to  varying  methods  of 
employment — in  this  case  different  airspace  coordina¬ 
tion  procedures  when  NEW  and  ESP  were  employed 
concurrently. 

INTEGRAL  FIRE  was  managed  using  an  inte¬ 
grated  product  team  and  senior  steering  group 
structure.  Team  leaders  were  responsible  to  the  overall 
event  leader  who  was  responsible  to  the  senior  steering 
group.  There  were  six  integrated  product  teams.  The 
analysis  team  translated  evaluation  objectives  into 
specific  data  requirements  and  refined  joint  operation 
contexts  and  conditions  under  which  the  data  needed 
to  be  collected.  The  LVC  team  defined  and  coordi¬ 
nated  distributed  components  to  assemble  the  joint 
mission  environment.  The  infrastructure  team  was 
responsible  for  all  technical  and  nontechnical  aspects  of 
the  networks  used  to  connect  LVC  components.  A 
security  team  coordinated  classification  guidance  and 
assisted  the  infrastructure  team  with  security  accredi¬ 
tation.  An  operations  team  was  responsible  for 
implementing  the  joint  operational  context  for  test 
activities,  including  specific  sequences  of  activities 
conducted  by  systems  under  test  and  supporting 


systems  of  systems  during  actual  testing.  Finally,  an 
integration  team  provided  facilitation  and  coordination 
among  the  other  teams.  All  six  teams  applied  various 
parts  of  aU  CTM  steps.  As  discussed  later,  this  has 
implications  regarding  future,  persistent  organizational 
structures  for  testing  in  a  joint  environment. 

Planning  and  coordination  across  distributed  IN¬ 
TEGRAL  FIRE  test  organizations  was  accomplished 
through  weekly  conference  calls  and  three  face-to-face 
planning  conferences.  Each  integrated  product  team 
conducted  its  own  conference  call  between  Monday 
and  Thursday.  Each  Friday,  all  team  leaders  partici¬ 
pated  in  a  conference  caU  with  the  integration  team 
and  event  leader  to  coordinate  actions  across  the 
integrated  product  team  structure.  The  event  leader 
also  facilitated  a  monthly  conference  call  composed  of 
senior  steering  group  members.  Face-to-face  confer¬ 
ences  brought  together  most  event  participants  (ap¬ 
proximately  100  test  engineers,  managers,  and  analysts 
from  21  difference  test  organizations)  to  bring 
everyone  to  a  common  understanding  of  overall 
planning  status  and  issues  and  to  conduct  detailed 
planning  and  integration  discussions.  In  terms  of  the 
JTEM  Capability  Test  Methodology,  the  initial 
planning  conference  focused  on  processes  in  CTM  1. 
The  midproject  and  final  planning  conferences  con¬ 
centrated  mostly  on  CTM  2  and  3,  respectively.  JTEM 
used  all  of  these  interactions  among  distributed 
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Figure  3.  CTM  1  Test  characterization  processes  used  during  INTEGRAL  FIRE. 


participants  to  evaluate  the  effectiveness  and  suitability 
of  CTM  version  1.0  processes,  as  well  as  to  assess  the 
applicability  of  the  INTEGRAL  FIRE  organizational 
structure  to  a  permanent  state  in  which  distributed 
tests  regularly  occur. 

Results  and  discussion 

CTM  1  processes,  shown  in  Figure  3,  were  accom¬ 
plished  before  and  during  the  initial  planning  confer¬ 
ence.  JTEM  personnel  provided  input  information  in 
the  form  of  a  test  concept,  joint  operational  context, 
and  an  evaluation  strategy  for  focused  developmental 
testing  in  a  joint  environment.  After  some  iterations 
and  negotiations  between  JTEM  and  the  integrated 
product  teams,  the  teams  completed  a  technical 
assessment  that  produced  an  initial  estimate  of  the 
distributed  joint  test  environment. 

For  example.  Figure  4a  shows  one  of  the  test- 
concept  depictions  used  in  early  program  introduction 
information  submitted  to  INTEGRAL  FIRE  teams 
for  technical  review.  With  assistance  from  JTEM 
engineers,  the  integrated  product  teams  took  this 
concept,  along  with  additional  information  about 
required  joint  operational  contexts  and  evaluation 
strategy,  and  produced  the  operational  view  shown  in 
Figure  4b.  The  LVC  team  also  produced  an  initial 
estimate  of  the  distributed  facilities  needed  to  assemble 
the  environment  shown  in  Figure  4,  and  the  operations 
team  developed  an  initial  sequence  of  actions  to  be 
accomplished  during  each  test  trial.  It  was  found  that 
current  CTM  1  processes  for  developing  joint  test 
concepts,  joint  operational  contexts,  and  joint  mission 
evaluation  strategies  are  too  important  to  be  confined 
to  test  characterization  by  distributed  test  organiza¬ 
tions.  Rather,  these  processes  should  be  accomplished 
when  acquisition  managers  are  preparing  the  overall 
test  and  evaluation  strategies  or  master  plans.  That 


way,  testing  in  a  joint  environment  can  be  integrated 
with  other  development  and  operational  testing 
throughout  acquisition  phases.  Another  important 
finding  was  that  none  of  the  integrated  product  teams 
could  produce  a  cost  estimate  for  distributed  tests.  This 
was  due,  at  least  partially,  to  a  lack  of  formal 
relationships  among  test  organizations  to  allow  for 
routine  distributed  testing.  Formal  agreements  wiU 
likely  also  be  needed  to  decide  which  test  organizations 
should  participate  in  distributed  testing  in  a  joint 
environment  for  any  particular  acquisition  program. 

CTM  2  processes  used  to  produce  a  test  plan  for 
INTEGRAL  FIRE  are  shown  in  Figure  5.  These 
processes  were  executed  iteratively  by  the  analysis, 
LVC,  and  operations  teams.  The  integration  team 
then  pulled  information  together  from  these  three 
teams  to  produce  an  actual  test  plan  document.  The 
midplanning  conference  was  the  primary  face-to-face 
meeting,  augmented  by  several  smaller  meetings  before 
and  afterward.  Figure  6  shows  two  example  outputs 
produced  by  CTM  2  processes.  Here,  the  operational 
architecture  in  Figure  4b  was  developed  into  an 
experimental  design  (Gray  2007)  and  a  detailed  test 
vignette  by  the  Develop  Test  Concept  process.  In 
addition.  Perform  LVC  Distributed  Environment 
Analysis  processes  produced  a  refined  LVC  design 
with  detailed  information  about  facilities  and  individ¬ 
ual  simulated  or  live  entities  located  at  each  facility.  In 
INTEGRAL  FIRE,  test  customers  (SAF/XC,  JSIC, 
and  JTEM)  approved  their  own  test  plans,  and 
participating  facilities  produced  and  approved  test 
plans  according  to  local  procedures.  We  concluded 
that  this  method  was  too  cumbersome  for  frequent 
distributed  testing  in  a  joint  environment.  Test 
organizations  should  consider  a  construct  in  which 
each  acquisition  program  has  a  lead  test  organization 
designated  for  distributed  testing.  Each  participating 
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Figure  4.  Inputs  and  outputs  of  technical  assessment  process  during  INTEGRAL  FIRE,  (a)  A  test-concept  depiction  used  as  input  to 
technical  assessment  process,  (b)  Operational  architecture  output  from  technical  assessment  process. 
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Figure  6.  Outputs  of  develop  test  design  process  during  INTEGRAL  FIRE,  (a)  Experimental  design  produced  by  the  develop  test 
design  process,  (b)  Vignette  details  produced  by  the  develop  test  design  process. 
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Figure  1.  CTM  3  implement  LVC  distributed  environment  processes  used  during  INTEGRAL  FIRE. 


test  organization  could  produce  a  test  plan  according  to 
local  formats  and  procedures,  with  added  sections  to 
accommodate  JTEM-recommended  methods  and  pro¬ 
cesses  for  distributed  testing  in  a  joint  environment. 
The  overall  distributed  test  plan  could  be  an  augment¬ 
ed  version  of  the  lead  test  organization’s  plan. 
Participating  test  organizations  could  approve  their 
own  plans,  while  the  lead  test  organization  commander 
would  provide  “distributed  approval”  by  mutual 
agreement.  Such  a  concept  might  better  suit  situations 
in  which  tests  in  a  joint  environment  are  integrated 
seamlessly  into  other  developmental  and  operational 
tests.  The  lead  test  organization  should  be  the 
organization  responsible  for  most  nondistributed 
testing  for  that  program. 

CTM  3  processes  used  to  produce  the  final  LVC 
distributed  environment  are  shown  in  Figure  7.  A  final 
planning  conference  and  two  week-long  distributed 
integration  periods  were  needed  to  execute  these 
processes.  Table  2  shows  a  simple  depiction  of  the 
LVC  distributed  configuration  produced  by  the  Design 
LVC  Distributed  Environment  Configuration  process. 
Figure  8  shows  the  LVC  distributed  environment 
produced  by  CTM  3  processes.  This  distributed 
environment  was  assembled  to  support  all  INTEGRAL 
LIRE  customers.  Customers  used  only  those  parts  of  the 
environment  needed  to  accomplish  their  individual  test 
objectives.  Specifically,  INTEGRAL  LIRE  was  able  to 
schedule  JSIC  testing  for  the  first  week  and  JTEM  and 
WWL  testing  during  different  times  the  second  week. 
This  is  directly  analogous  to  configuring  an  open-air  test 
range  to  accommodate  a  set  of  test  customers,  then 


Table  2.  LVC  distributed  configuration  produced  by  design 
LVC  distributed  environment  configuration 


Function 

Primary  configuration 

Backup  configuration 

JTAC 

WSMR 

Eglin 

NEW 

SIMAF 

Eglin 

Launch  aircraft 

SIMAF 

Eglin 

NEW  targets 

WSMR 

SIMAF 

CAOC 

Langley 

Langley 

ESP 

Redstone 

Redstone 

scheduling  various  parts  of  the  range  separately  or 
concurrently  to  conduct  each  customer’s  testing.  But 
distributed  testing,  such  as  in  INTEGRAL  LIRE, 
requires  multiple  test  facilities  to  be  set  up  and  managed 
as  though  they  were  a  single  test  facility.  The  good  news 
is  that  much  of  the  work  producing  CTM  3  products 
was  simply  due  to  the  lack  of  a  persistent  distributed  test 
environment  and  standing  organizational  relationships 
to  manage  that  environment.  INTEGRAL  LIRE  made 
significant  progress  toward  persistence.  Test  organiza¬ 
tions  should  consider  establishing  formal  relation¬ 
ships — the  integrated  product  team  structure  used  in 
INTEGRAL  LIRE  is  a  sensible  place  to  start — so  that 
the  distributed  test  environment  can  be  managed  as  a 
single  facility.  Evidence  suggests  that  such  things  as 
standard  data  products,  permanent  configuration  con¬ 
trol,  and  a  full-time  verification  and  validation  group 
would  have  substantially  reduced  the  effort  needed  to 
assemble  the  INTEGRAL  EIRE  test  environment. 

Conclusion 

Substantial  improvements  were  made  to  our  methods 
and  processes  by  having  experienced  testers  use  them  to 
plan  and  conduct  actual  test  activity.  Processes  currently 
in  CTM  1  for  developing  joint  test  concepts,  joint 
operational  contexts,  and  joint  mission  evaluation 
strategies  were  found  to  be  too  important  to  be  confined 
to  test  characterization  by  distributed  test  organizations. 
Figure  9  shows  CTM  version  1.1  in  which  these 
processes  are  moved  to  a  sixth  step,  CTM  0,  for 
acquisition  managers  to  prepare  overall  test  and 
evaluation  strategies  and  master  plans.  A  lack  of 
persistent  formal  relationships  among  test  organizations 
led  to  problems  with  cost  estimation  and  increased 
workload  in  assembling  a  distributed  joint  test  environ¬ 
ment.  Implementation  of  test  planning  processes  during 
INTEGRAL  FIRE  was  too  cumbersome  for  frequent 
distributed  testing  in  a  joint  environment.  As  a  result, 
we  recommend  test  organizations  consider  a  construct 
where  each  acquisition  program  has  a  lead  test 
organization  designated  for  distributed  testing.  We  also 
recommend  test  organizations  consider  establishing 
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Figure  8.  LVC  distributed  environment  produced  by  integrate  LVC-DE  process. 
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Figure  9.  Capabiiity  Test  Methodology  version  1. 1  refiecting  iessons  learned  during  INTEGRAL  FIRE. 
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formal  relationships  so  that  the  distributed  test 
environment  can  be  managed  as  a  single  facility.  JTEM 
released  version  2.0  of  the  Capability  Test  Methodology 
in  early  2008  and  used  those  updated  processes  to  plan 
and  conduct  another  set  of  distributed  tests.  The 
department’s  ultimate  goal  is  to  test  and  evaluate 
requirements  as  part  of  the  overarching  acquisition 
process  in  realistic  joint  mission  environments  and 
institutionalize  testing  in  a  joint  environment.  □ 
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